ESQL: Add url_encode function #133494

mouhc1ne · 2025-08-25T19:58:15Z

Implement url_encode function

github-actions · 2025-08-25T20:00:10Z

🔍 Preview links for changed docs

docs/reference/query-languages/esql/kibana/docs/functions/url_encode.md

mouhc1ne · 2025-08-25T20:02:11Z

How the new function looks in the docs.

mouhc1ne · 2025-08-25T20:29:36Z

...src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/UrlEncode.java

+    @ConvertEvaluator()
+    static BytesRef process(final BytesRef val) {
+        String s = val.utf8ToString();
+        String encoded = URLEncoder.encode(s, StandardCharsets.UTF_8);


Two concerns I have about this snippet:

This actually over-encodes the input. For example http://elastic.co becomes http%3A%2F%2Felastic.co. I wonder if we have a better option internally.

Performance when passing strings around, as I see all other evaluators directly manipulating byte arrays.

2. I see all other evaluators directly manipulating byte arrays.

Many don't. We'd prefer if they did, but sometimes that's life.

I think the over-encoding is a bigger deal - we should figure out what the folks who asked for this expect from it. If you add this as a SNAPSHOT function then we have a lot of latitude to change it later.

http://elastic.co becomes http%3A%2F%2Felastic.co

From what I read in the RFC, those are reserved characters and "should" be encoded. So I would say that it's fine (?)

nik9000 · 2025-08-25T21:58:16Z

...src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/UrlEncode.java

+    @ConvertEvaluator()
+    static BytesRef process(final BytesRef val) {
+        String s = val.utf8ToString();
+        String encoded = URLEncoder.encode(s, StandardCharsets.UTF_8);


2. I see all other evaluators directly manipulating byte arrays.

Many don't. We'd prefer if they did, but sometimes that's life.

I think the over-encoding is a bigger deal - we should figure out what the folks who asked for this expect from it. If you add this as a SNAPSHOT function then we have a lot of latitude to change it later.

...est/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/UrlEncodeTests.java

ivancea

LGTM!

docs/changelog/132869.yaml

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

...est/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/UrlEncodeTests.java

ivancea · 2025-08-26T10:47:45Z

...src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/UrlEncode.java

+    @ConvertEvaluator()
+    static BytesRef process(final BytesRef val) {
+        String s = val.utf8ToString();
+        String encoded = URLEncoder.encode(s, StandardCharsets.UTF_8);


http://elastic.co becomes http%3A%2F%2Felastic.co

From what I read in the RFC, those are reserved characters and "should" be encoded. So I would say that it's fine (?)

x-pack/plugin/esql/qa/testFixtures/src/main/resources/string.csv-spec

docs/reference/query-languages/esql/_snippets/lists/string-functions.md

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

elasticsearchmachine · 2025-08-27T19:44:08Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-08-27T19:44:08Z

Pinging @elastic/kibana-esql (ES|QL-ui)

nik9000

Looks good to me too.

mouhc1ne · 2025-08-28T14:51:11Z

The failing release-test (https://buildkite.com/elastic/elasticsearch-pull-request/builds/88528/steps/canvas?jid=0198f0c9-4928-4cf3-9939-0a867d857f5f) is unrelated to these changes. Removing test-release label.

- Rename the capability to exclude the decoding portion, since this PR is only concerned with encoding. - Mark url_encode as SNAPSHOT to allow changes in encoding logic. - Add tests that use random strings to test a wide range of characters, that run alongside the ones that use random URLs. - Minor change in the sample csv test string to showcase a fictional URL with path and few query params.

- Remove url_encode from docs since it's a snapshot function - Declare it along other snapshot functions in EsqlFunctionRegistry - Annotate UrlEncode similarly to other snapshot functions - Add url_encode capability to csv tests

- Add url_decode as a snapshot function. - Move common logic to both encode/decode to a parent class

* ESQL: Add url_encode function

- Add url_decode as a snapshot function. - Move common logic to both encode/decode to a parent class

BASE=37f65b0bf8eb0f75ea696ad136eac0bd50005330 HEAD=7572fc5993ae95ef569fd2283666951cb387ff78 Branch=main

elasticsearchmachine added the v9.2.0 label Aug 25, 2025

mouhc1ne force-pushed the url_encode_function branch from da1095a to 1c7be9e Compare August 25, 2025 20:12

mouhc1ne commented Aug 25, 2025

View reviewed changes

mouhc1ne added >enhancement :Analytics/ES|QL AKA ESQL labels Aug 25, 2025

nik9000 reviewed Aug 25, 2025

View reviewed changes

ivancea self-requested a review August 26, 2025 09:30

ivancea reviewed Aug 26, 2025

View reviewed changes

mouhc1ne added ES|QL-ui Impacts ES|QL UI test-release Trigger CI checks against release build labels Aug 26, 2025

mouhc1ne force-pushed the url_encode_function branch 2 times, most recently from 4282895 to a3f01fb Compare August 26, 2025 21:17

mouhc1ne requested review from ivancea and nik9000 August 26, 2025 22:58

ivancea reviewed Aug 27, 2025

View reviewed changes

docs/reference/query-languages/esql/_snippets/lists/string-functions.md Outdated Show resolved Hide resolved

ivancea reviewed Aug 27, 2025

View reviewed changes

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java Outdated Show resolved Hide resolved

mouhc1ne force-pushed the url_encode_function branch from a3f01fb to b24117f Compare August 27, 2025 11:55

mouhc1ne requested a review from ivancea August 27, 2025 11:58

mouhc1ne force-pushed the url_encode_function branch 4 times, most recently from c6fb600 to 25545d4 Compare August 27, 2025 19:32

mouhc1ne marked this pull request as ready for review August 27, 2025 19:43

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Aug 27, 2025

nik9000 approved these changes Aug 27, 2025

View reviewed changes

mouhc1ne force-pushed the url_encode_function branch from 25545d4 to 2d3cdcc Compare August 28, 2025 13:04

mouhc1ne removed the test-release Trigger CI checks against release build label Aug 28, 2025

mouhc1ne added 3 commits August 28, 2025 10:51

ESQL: Add url_encode function

561da2e

ESQL: Add url_encode function

7572fc5

- Remove url_encode from docs since it's a snapshot function - Declare it along other snapshot functions in EsqlFunctionRegistry - Annotate UrlEncode similarly to other snapshot functions - Add url_encode capability to csv tests

mouhc1ne force-pushed the url_encode_function branch from 2d3cdcc to 7572fc5 Compare August 28, 2025 14:52

ivancea approved these changes Aug 28, 2025

View reviewed changes

mouhc1ne merged commit 875e6f3 into elastic:main Aug 28, 2025
33 checks passed

mouhc1ne added a commit to mouhc1ne/elasticsearch that referenced this pull request Aug 28, 2025

ESQL: Add url_decode function (elastic#133494)

335bbfb

- Add url_decode as a snapshot function. - Move common logic to both encode/decode to a parent class

mouhc1ne added a commit to mouhc1ne/elasticsearch that referenced this pull request Aug 29, 2025

ESQL: Add url_decode function (elastic#133494)

4b80e43

- Add url_decode as a snapshot function. - Move common logic to both encode/decode to a parent class

mouhc1ne added a commit to mouhc1ne/elasticsearch that referenced this pull request Aug 29, 2025

ESQL: Add url_decode function (elastic#133494)

1ce7951

- Add url_decode as a snapshot function. - Move common logic to both encode/decode to a parent class

mouhc1ne added a commit to mouhc1ne/elasticsearch that referenced this pull request Aug 29, 2025

ESQL: Add url_decode function (elastic#133494)

c3b9ee5

JeremyDahlgren pushed a commit to JeremyDahlgren/elasticsearch that referenced this pull request Aug 29, 2025

ESQL: Add url_encode function (elastic#133494)

0bc5e76

* ESQL: Add url_encode function

mouhc1ne added a commit to mouhc1ne/elasticsearch that referenced this pull request Aug 29, 2025

ESQL: Add url_decode function (elastic#133494)

de50585

- Add url_decode as a snapshot function. - Move common logic to both encode/decode to a parent class

mouhc1ne added a commit to mouhc1ne/elasticsearch that referenced this pull request Aug 29, 2025

ESQL: Add url_decode function (elastic#133494)

695cbb7

mouhc1ne added a commit to mouhc1ne/elasticsearch that referenced this pull request Aug 29, 2025

ESQL: Add url_decode function (elastic#133494)

275a53d

mouhc1ne added a commit that referenced this pull request Sep 2, 2025

ESQL: Add url_decode function (#133494) (#133761)

88f55db

mouhc1ne self-assigned this Sep 3, 2025

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 2, 2025

Mirror upstream elastic#133494 as single snapshot commit for AI review

b455a30

BASE=37f65b0bf8eb0f75ea696ad136eac0bd50005330 HEAD=7572fc5993ae95ef569fd2283666951cb387ff78 Branch=main

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 6, 2025

Mirror upstream elastic#133494 as single snapshot commit for AI review

3f988a4

BASE=37f65b0bf8eb0f75ea696ad136eac0bd50005330 HEAD=7572fc5993ae95ef569fd2283666951cb387ff78 Branch=main

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 8, 2025

Mirror upstream elastic#133494 as single snapshot commit for AI review

67eb297

BASE=37f65b0bf8eb0f75ea696ad136eac0bd50005330 HEAD=7572fc5993ae95ef569fd2283666951cb387ff78 Branch=main

phananh1010 added a commit to phananh1010/elasticsearch that referenced this pull request Oct 17, 2025

Mirror upstream elastic#133494 as single snapshot commit for AI review

e38b684

BASE=37f65b0bf8eb0f75ea696ad136eac0bd50005330 HEAD=7572fc5993ae95ef569fd2283666951cb387ff78 Branch=main

ESQL: Add url_encode function #133494

ESQL: Add url_encode function #133494

Uh oh!

Conversation

mouhc1ne commented Aug 25, 2025

Uh oh!

github-actions bot commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

mouhc1ne commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mouhc1ne Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

nik9000 Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

ivancea Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

nik9000 Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ivancea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivancea Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 27, 2025

Uh oh!

elasticsearchmachine commented Aug 27, 2025

Uh oh!

nik9000 left a comment

Choose a reason for hiding this comment

Uh oh!

mouhc1ne commented Aug 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Aug 25, 2025 •

edited

Loading

mouhc1ne commented Aug 25, 2025 •

edited

Loading